Analysis of Information Management and Scheduling Technology in Hadoop

Development of big data computing has brought many changes to society and social life is constantly digitized. ‘How to handle vast amounts of data’ has become a more and more fashionable topic. Hadoop is a distributed computing software framework, which includes HDFS and MapReduce distributed computing method, make distributed processing huge amounts of data possible. Then job scheduler determines the efficiency of Hadoop clusters and user experience. Under the premise of familiar with the mechanism of Hadoop’s running tasks, make a full analysis of the existing Hadoop task scheduling algorithm, such as FIFO-Scheduler, Capacity –Scheduler, FairShare-Scheduler and LATE-Scheduler, found that the existing scheduling algorithms do not perceive the performance of the computing node, so that it cannot assign different tasks depending on the machine performance in heterogeneous Hadoop cluster. Subject Categories and Descriptors H.2.8 [Database Applications]: Data Mining; H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval General Terms: Data Mining, Data Clusters, Hadoop